{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb0675a7",
   "metadata": {},
   "source": [
    "# Relevance Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use the Relevance evaluator to assess the relevance of AI-generated responses. The evaluator supports multiple input formats including:\n",
    "- Simple query and response pairs\n",
    "- Multi-turn conversations\n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure OpenAI Endpoint to be used for evaluation\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure OpenAI Key to be used for evaluation\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure OpenAI API version to be used for evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a21aa7e",
   "metadata": {},
   "source": [
    "## What is Relevance?\n",
    "\n",
    "The Relevance evaluator assesses the ability of AI responses to capture the key points and address the user's query appropriately. It measures how well the response relates to and answers the specific question or request.\n",
    "\n",
    "**Scoring:** Relevance scores range from 1 to 5, with:\n",
    "- **1**: Completely irrelevant to the query\n",
    "- **2**: Partially relevant but misses key points\n",
    "- **3**: Moderately relevant with some key points addressed\n",
    "- **4**: Highly relevant with most key points covered\n",
    "- **5**: Perfectly relevant, fully addressing the query\n",
    "\n",
    "High relevance scores indicate the AI system's understanding of the input and capability to produce contextually appropriate outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0dd0ebf",
   "metadata": {},
   "source": [
    "## Relevance Evaluator Input Requirements\n",
    "\n",
    "The Relevance evaluator supports the following input formats:\n",
    "\n",
    "1. **Simple Query-Response Evaluation:**\n",
    "   - `query`: The user's question or request (str)\n",
    "   - `response`: The AI's response to evaluate (str)\n",
    "\n",
    "2. **Conversation Evaluation:**\n",
    "   - `conversation`: Multi-turn conversation with messages list and optional context (Conversation object)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "519631da",
   "metadata": {},
   "source": [
    "### Initialize Relevance Evaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3484e790",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation import RelevanceEvaluator, AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "# Configure the model\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "# Initialize the evaluator\n",
    "relevance_evaluator = RelevanceEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f739700",
   "metadata": {},
   "source": [
    "## Sample Evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83d8d0ec",
   "metadata": {},
   "source": [
    "### Query and Response as Strings (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43eaf25d",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "response = \"The current weather in Seattle is rainy with a temperature of 14°C. It's typical Pacific Northwest weather for this time of year with overcast skies and light precipitation.\"\n",
    "\n",
    "result = relevance_evaluator(query=query, response=response)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89266048",
   "metadata": {},
   "source": [
    "### Example of Irrelevant Response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69761825",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Irrelevant response example\n",
    "query = \"What are the symptoms of flu?\"\n",
    "response = \"The stock market had significant fluctuations today, with technology stocks leading the decline.\"\n",
    "\n",
    "result = relevance_evaluator(query=query, response=response)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd158e43",
   "metadata": {},
   "source": [
    "### Conversation as Dict with Messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf17774b",
   "metadata": {},
   "outputs": [],
   "source": [
    "conversation = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"I'm planning a trip to Japan. Can you help me with some recommendations?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"I'd be happy to help you plan your trip to Japan! Here are some popular destinations: Tokyo for modern city life and culture, Kyoto for traditional temples and gardens, Osaka for amazing food, and Mount Fuji for scenic beauty. What type of experiences are you most interested in?\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"I'm particularly interested in traditional culture and temples.\"\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"Perfect! For traditional culture and temples, I highly recommend Kyoto. Visit Fushimi Inari Shrine with its thousands of red torii gates, Kinkaku-ji (Golden Pavilion), and Gion district where you might spot geishas. Nara is also wonderful with its deer park and Todai-ji Temple housing a giant Buddha statue.\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "result = relevance_evaluator(conversation=conversation)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
